Moving image generation apparatus, moving image playback apparatus, their control method, and storage medium

ABSTRACT

The invention generates moving image information obtained by compositing a moving image of a user and objects selected by the user, and provides the generated moving image information to the user. For instance, when the user has inserted a coin or coins into a device, a select menu of pre-stored stories is displayed, and the user selects one story from the menu. Then, the user selects a desired background scene and feature player in that story to determine pre-stored objects. After that, an image of the user is sensed, and is stored as a moving image object. The moving image object is then composited with the previously selected objects according to MPEG 4 to generate single stream information. The generated moving image information is then delivered to the user.

FIELD OF THE INVENTION

The present invention relates to an apparatus for generating a movingimage the user wants, a playback apparatus, their control method, and astorage medium.

BACKGROUND OF THE INVENTION

A print system which composites an image obtained by sensing a user whoinserted a coin or coins, and an image prepared in advance, and printsthe composited image as a sticker is known. As such system, JapanesePatent Laid-Open Nos. 04-104572 or 04-506264 is available. Such systembasically stores some background images, and comprises a camera fortaking a picture, a computer for controlling composition of the sensedimage with one background image selected from the plurality ofbackground images, image sensing of the camera, and an image printprocess (to be described below), and a printer for printing out an imageobtained by compositing the sensed image and the selected backgroundimage. However, these systems and patents are not designed for a movingimage.

A system that expands the above system to process a movie-like image isdisclosed in Japanese Patent Laid-Open No. 10-115866 or 10-117344.However, the “movie-like image” in such system is a series of photos orpanel which can be printed out by a printer, and several still imagesare merely combined using the aforementioned mechanism. Hence, suchsystem is not designed for an actual moving image.

The reasons why the conventional systems are not designed for a movingimage, i.e., problems to be solved upon building a system for a movingimage, are as follows.

1) A moving image has a considerably larger information size than astill picture, and the apparatus components such as a memory, imageprocessing means, and the like cannot be efficiently implemented.

2) A still-image can easily undergo an image process such as compositionbetween a background image and sensed image, but means for editing amoving image by a simple process is not easily available (except forexpensive means such as editing apparatuses for TV, movie, and thelike).

3) A moving image is hard to output as a printed image, and is notsuitable as a medium that everyone can use.

4) The user can exchange printed images output as stickers with friends,but cannot do so for a moving image due to its large information size.

Even when a moving image is allowed to be easily sent as digital data,the following problems concerning copyrights are additionally posed.

5) When an output is printed matter, it is rarely illicitly copied(image quality inevitably deteriorates even by color copying or thelike), but when an output is digital data, copies that are free fromdeterioration of image quality can be easily formed. If datacorresponding to a background scene or feature player is copyrighted, itis difficult to attain copyright protection.

6) Even a copyright-free movie of a certain user violates the right oflikeness or privacy of that user if it is illicitly copied and sent.

7) If an output is a sticker or the like, a fee can be charged for oneset of a predetermined number of stickers. However, when a digital dataoutput is used to implement the same function, a copy count must becontrolled, and a charging system corresponding to the copy count mustbe built, thus requiring a new technique.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and has as its object to provide a moving image generationapparatus which can offer moving image information services to theusers, a moving image playback apparatus, their control method, and astorage medium.

In order to achieve the above object, for example, a moving imagegeneration apparatus of the present invention comprises the followingarrangement.

That is, a moving image generation apparatus comprises:

-   -   storage means for storing a plurality of compressed objects        which specify stream information with respect to a time axis;    -   input means for inputting video data of a moving image obtained        by predetermined image sensing means as the object; and    -   output means for multiplexing at least one desired object of the        objects stored in the storage means and the object input by the        input means and outputting the multiplexed objects as single        stream information.

It is another object of the present invention to provide a moving imageoutput apparatus which controls output of objects contained in a scene,and plays back a bitstream including that scene, a moving image playbackapparatus, a portable terminal, their control method, and a storagemedium.

In order to achieve the above object, for example, a moving image outputapparatus of the present invention comprises the following arrangement.

That is, a moving image output apparatus comprises:

-   -   storage means for storing various object data;    -   input means for inputting moving image data sensed by        predetermined image sensing means;    -   information generation means for generating information data        that pertains to a user condition of the object data; and    -   output means for outputting stream data by multiplexing the        object data stored in the storage means, the moving image data        input by the input means, and the information data generated by        the information generation means.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an amusement apparatus according to thefirst embodiment of the present invention;

FIG. 2 is a flow chart showing the operation processing sequence of theamusement apparatus of the first embodiment;

FIG. 3 is a diagram for explaining MPEG-4;

FIG. 4 is a block diagram of an MPEG-4 player;

FIG. 5 is a block diagram of a portable terminal according to theembodiment of the present invention;

FIG. 6 is a flow chart showing the processing sequence of the portableterminal of the first embodiment;

FIG. 7 shows an example of system arrangement of the first embodiment;

FIG. 8 shows another example of system arrangement of the firstembodiment;

FIG. 9 shows still another example of system arrangement of the firstembodiment;

FIG. 10 is a block diagram of a processor of the first embodiment;

FIG. 11 shows an example of a story select menu in the first embodiment;

FIG. 12 shows an example of a background select menu in the firstembodiment;

FIG. 13 shows an example of a feature player select menu in the firstembodiment;

FIG. 14 shows an example of the data structure stored in a storage unit105 of the first embodiment;

FIG. 15 shows an example of an email address input window in the secondembodiment of the present invention;

FIG. 16 is a flow chart showing the DB update operation processingsequence of an apparatus in the fourth embodiment of the presentinvention;

FIG. 17 is a block diagram of a moving image output apparatus accordingto the fifth embodiment of the present invention;

FIG. 18 is a flow chart showing the operation flow of the moving imageoutput apparatus of the fifth embodiment;

FIG. 19 is a block diagram of an MPEG-4 player in the fifth embodiment;

FIG. 20 is a block diagram of a portable terminal in the fifthembodiment;

FIG. 21 is a flow chart showing the operation flow of the portableterminal in the fifth embodiment;

FIG. 22 is a block diagram of a copyright protection unit of a movingimage processing apparatus of the fifth embodiment; and

FIG. 23 is a block diagram of a copyright protection unit of a portableterminal of the fifth embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described indetail hereinafter with reference to the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram of a moving image amusement apparatusaccording to the present invention, and FIG. 2 shows its operation flowchart. Note that a program associated with the flow chart in FIG. 2 isstored in a storage means 105 (to be described later), and is loadedonto a RAM (not shown) upon execution.

The user inserts a coin or coins into an insertion port 101 (step S201).Note that the present invention is not limited to coin, but a creditcard, or an IC card, prepaid card, or the like having a function ofelectronically receiving money may be used.

Upon detection of coins in a predetermined amount, a display 104displays stories prepared in advance in a storage unit 105 (e.g., astorage device such as a hard disk, MO, CD, DVD, or the like) as someselectable menu items via a processor 106 (to be described later), andthe user selects a desired story at a console 103 (step S202).

After the story is selected, some background scenes, feature players,and the like corresponding to that story, which are stored in thestorage unit 105, are displayed on the display screen of the display 104via the processor 106, and the user selects a desired background scene,feature player, and the like at the console 103 (step S203). In place ofdisplaying the stories, background scenes, and the like, they may beannounced by voice using a speaker (or earphone) 107, and the user mayinput his or her choice via a microphone 108.

After the story, background scene, feature player, and the like aredetermined, and when the user inputs a start instruction at the console103, the motion and voice of the user corresponding to the story arecaptured as digital moving image information using a camera 102 and themicrophone 108. The captured digital information is extracted andcompressed by the processor 106 (step S204). After an elapse of apredetermined period of time, the capture process is complete, and theprocessor 106 composites the compressed sensed information with theselected background scene and feature player and plays back theinformation using the display 104 and speaker 107 (step S205). Theapparatus prompts the user to check if the playback information is OK.If the user inputs a re-input instruction at the console 103, theprocess can be repeated from step S202, S203, or S204 (step S206). Notethat this repetition process may be omitted, and the process may be doneonly once. Alternatively, the flow may return to step S201, and the usermay select a repetition process by inserting extra money.

If the user inputs a confirmation instruction of the playback result,the playback information is transmitted using a communication unit 109(step S207). As the communication unit 109, various devices areavailable. For example, since portable terminals (note type PC, PDA, andthe like) have prevailed recently, and most of them have wired/wirelesscommunication interfaces such as USB (Universal Serial Bus), IrDA(Infrared Data Association), and the like, a moving image composited bythis apparatus is sent as a file to the user's device using suchinterface.

In the above description, the respective units 101 to 109, and theoperation shown in FIG. 2 are controlled by a controller 110 whichcomprises a CPU, a ROM/RAM for storing programs, and the like.

The processor 160 (including the controller 110) will be described indetail below. In this system, data such as stories, background scenes,feature players, a sensed moving image, audio, and the like (pre-storedin a storage medium such as a hard disk, CD, or the like) areindividually processed as compressed objects. These objects arecompressed/expanded, composited, and played back when they are used. Asa mechanism for implementing a composition process of a plurality ofobjects, MPEG-4 (Moving Picture Experts Group Phase4) which isstandardized by ISO can be used.

MPEG-4 is a scheme for encoding moving image data, audio data, and thelike, handling individual encoded data as objects, combining theseso-called multimedia data, and sending them as a single bitstream. Sinceindividual data are handled as objects, the received bitstream can beeasily decomposed into objects, such as audio objects, video object,text objects, and the like, and can be reassembled. FIG. 3 illustratesthis process. In FIG. 3, blocks for encoding respective objects (301 to305 in FIG. 3), and multiplexing the encoded objects (306) correspond tothe apparatus of this embodiment, and blocks for demultiplexing themultiplexed stream (finally saved as a file in the user's terminal)(307), and decoding individual objects (308 to 312) and composition unit313 for composing the decoded objects correspond to software of theuser's terminal.

As a scheme for compositing these objects to obtain a given scene, BIFS(Binary Format For Scenes) obtained by modifying VRML (Virtual RealityModeling Language) is known. This BIFS is called scene descriptioninformation and is described in binary, and a scene is compositedaccording to BIFS. Since individual objects required for such scenecomposition independently undergo optimal encoding and are sent, thedecoding side (user's terminal) independently decodes objects, andcomposites and outputs a scene by synchronizing the time axes of theindividual data with that inside a player in accordance with thedescription of BIFS.

FIG. 4 shows a schematic arrangement of a general MPEG-4 player. Atransmission path 401 is a data path such as various networks, computerbus, and the like, and is a network to which an MPEG-4 stream is input.Note that the transmission path also means an interface between theplayer and a recording medium device such as a CD-ROM, DVD-ROM, DVD-RAM,or the like in addition to the communication path. In the player, anMPEG-4 stream delivered from the network or played back from therecording medium device is input to a demultiplexer 402. The MPEG-4stream is demultiplexed by the demultiplexer 402 into scene descriptiondata, moving image object data, audio object data, object descriptiondata, and the like, which are respectively input to corresponding synclayer blocks (buffer memories) 403. Note that the audio object data hasundergone high-efficiency (compression) coding such as known CELP (CodeExcited Linear Prediction) coding, transform-domain weighted interleavevector quantization (TWINVQ) coding, or the like. Also, the moving imageobject data has undergone high-efficiency coding by, e.g., known MPEG-2,H-263, or the like.

The respective object data of the sync layer blocks 403 are input tocorresponding decoders 404 to 407. The decoders 404 to 407 decode theaforementioned scene description data, moving image object data, audioobject data, and the like which have undergone high-efficiency coding.In FIG. 4, the MPEG-4 bitstream contains a plurality of different typesof objects in each of audio objects, moving image objects, and objectdescription data since this embodiment assumes an apparatus which candecode these objects. For this reason, a plurality of sets of sync layerblocks and decoders 404 to 407 are prepared in correspondence with audioobjects, moving image objects, and object description data, but they canbe modified depending on the systems used.

The audio objects, moving image objects, and object description datadecoded by the decoders 405 to 407 undergo a composition/graphic processin a scene composition unit 408 on the basis of the scene descriptiondata decoded by the scene description decoder 404. A final data sequenceobtained in this way is supplied to an output device 409 such as adisplay, printer, or the like, and is visualized and made audible.

Hence, the stories stored in the storage unit 105 correspond to scenedescription data of BIFS, and background scenes, feature players, andthe like are managed as compressed objects in units of data (scenedescription data is also managed as one compressed object). For thisreason, even a moving image requires a small information size to bestored.

The processor 106 has an arrangement shown in FIG. 10. The imagesensing/compression process (step S204) in FIG. 2 is executed by animage extraction unit 1002 and compression unit 1003. An imageextraction scheme is not specified in MPEG-4. For example, theextraction unit 1002 may comprise means for implementing a knownchroma-key separation method that sets the background in specific color(blue back or the like), and separating pixels of that color as thoseoutside an object. The compression unit 1003 is implemented by theaforementioned MPEG-2 moving image encoder, CELP audio encoder, and thelike, thus outputting video and audio objects.

The composition/playback process (step S205) is executed by a scenecomposition unit 1001. This scene composition unit 1001 comprises thecomponents 403 to 408 shown in FIG. 4. That is, the respective objectsstored in the storage unit 105 and the compressed sensed moving imageare individually input to the sync layer blocks 403, and undergodecoding corresponding to individual encoding schemes of scenedescription, audio, moving image, and the like. The decoded objects arecomposited into a scene by the scene composition unit 408, and areoutput as signals to the display 104, speaker 107, and the like.

In this embodiment, objects that explain stories, and objects indicatingselectable background scenes, feature players, and the like (steps S202and S203) are prepared in advance, and information obtained by capturingand encoding a video sensed by the camera 102 is handled as one object.That is, these various objects can be implemented by the scenecomposition unit 1001. The last transmission process (step S207) canindividually send objects, but can easily send a story and a pluralityof objects based on that story as an MPEG-4 stream using the multiplexer306 in FIG. 3 like 1004 in FIG. 10.

Upon executing composition/playback process (step S205), various effectssuch as fade-out, color change, and the like can be added by a knownedit process.

FIG. 5 is a block diagram of the user's portable terminal in thisembodiment, and FIG. 6 shows its operation flow. This portable terminalhas a communication function of receiving objects created by theaforementioned process, and also a function of allowing the user toindividually enjoy contents by compositing and playing them back.

The portable terminal of this embodiment can display some processingmenu items on a display 504, and the user can select a process using aconsole 503. The menu includes a process for sending/receiving objects,a process for compositing/playing back a moving image, a process forchanging each object, and the like.

The user selects a process from the menu displayed on the display 504(step S602). Assume that the user selects the transmission/receptionprocess.

In this case, the flow advances to step S605, and objects are receivedfrom the aforementioned amusement apparatus of this embodiment oranother portable terminal using a communication unit 501. In this step,a process for transmitting objects to another terminal is done. Assumethat objects (including BIFS corresponding to a story) created byanother apparatus/terminal are received. This transmission/receptioninstruction is issued at the console 503, and objects received via thecommunication unit 501 are stored in a storage unit (a storage devicesuch as a hard disk or the like) 505. In the transmission process,objects stored in the storage unit 505 are designated using the display504 and console 503, and are transmitted to another portable terminalvia the communication unit 501. In order to confirm objects to betransmitted/received, the video and audio of those objects may be playedback via a processor 506 (to be described later), and may be stored ortransmitted after confirmation.

If the user selects the composition/playback process, the flow advancesto step S604, and objects which are stored in the storage unit 505 andform the story are played back via the processor 506 using the display504 or a speaker 502. The processor 506 can be implemented by the scenecomposition unit 1001 in FIG. 10.

If the user selects the object change process, the flow advances to stepS603. In this step, by adding objects via a communication with anotherterminal in step S605, objects contained in one story stored in thestorage unit 505 may be replaced by those added objects, or thoseobjects may be added to the story. Such process is implemented byrewriting information called a descriptor which indicates the attributeof each object or information described in BIFS within an allowablerange.

It is then checked if the process is to end (step S606). For example,after some objects are received from the amusement apparatus of theembodiment mentioned above or another portable terminal and are storedby the transmission/reception process in step S605, the process mayproceed to confirm the contents of the received objects by selecting thecomposition/playback process in step S604. Furthermore, the process mayproceed to repeat the composition/playback process in step S604 whileexecuting the object change process in step S603. Finally, the processends.

When information received from the amusement apparatus or the like is anMPEG-4 stream multiplexed by a multiplexer 306 in FIG. 3, the processor506 include the demultiplexer 402. The outputs from the demultiplexerare supplied not to the sync layer blocks but to the storage unit 505. Acontroller 507, which comprises a CPU, a ROM/RAM for storing programs,and the like, controls the respective units 501 to 506, and theoperation shown in FIG. 6.

As described above, according to the first embodiment, a moving image ofthe user can be provided together with the background scene and featureplayer designated by the user in accordance with the story of user'schoice.

The user's portable terminal need not be a special one and can use aversatile information processing apparatus (personal computer or thelike), since it can be implemented by software as long as it comprises acommunication means.

A moving image amusement system using the amusement apparatus, portableterminal, and the like of the above embodiment will be explained belowusing FIGS. 7 to 9.

Devices 701 to 710 shown in FIG. 7 respectively correspond to 101 to 110shown in FIG. 1. Note that a storage unit, processor, and controllercorresponding to 705, 706, and 710 are built in that apparatus. A box720 houses an amusement apparatus, and has an interior painted toprovide a blue back, and a door (not shown) through which the userenters the box. The blue back is one scheme for extracting objects, andwhen the moving image of the user in the box is sensed by the camera702, it is captured as one object that does not contain any images otherthan that of the user. Note that the component 720 need not have a boxshape, but may be a curtain or the like that can provide only a blueback background. When the user has inserted a coin or coins and madepredetermined operation, recording is done for a predetermined period oftime. The sensed video is displayed on the display device 704 in realtime, and the user can confirm it. At this time, image sensingconditions (story, feature player, background) the user sets aredisplayed together.

An example of the operation is as follows.

When the user has inserted a coin, a menu shown in FIG. 11 is displayed.The user selects a desired story (situation) using the console 103. In adefault state, the uppermost item (“golf lesson” in FIG. 11) isselected, and the cursor can move to other items by operating an up ordown key on the console 103. When the user has selected a desired itemand pressed an OK key on the console 103, a screen shown in FIG. 12 isdisplayed. In this case, a background scene is selected in practice. Ifthe selected story is “golf lesson”, a list of background scenesassociated with golf are displayed, and the user selects one of theseimages. After that, a feature player select screen is displayed, asshown in FIG. 13. The user selects a favorite feature player on thisscreen. When the user has selected the background scene and featureplayer, and instructed start of recording on the console 103, imagesensing of the camera 102 starts, and the playback image of the selectedfeature player is displayed on the selected background scene. The usertalks or acts in correspondence with the display. Note that the imagesensing time is fixed, and the remaining time is displayed on thedisplay screen. In this way, the user can recognize the remaining time.The image of the user is extracted from an image sensed by the camera102 and including other portions by a known chroma-key separationmethod, and the extracted image is saved as an object.

The object as the image of the user is handled as one object in a scenedescription file prepared in advance in association with the story. Thatis, the display position of the user is determined depending on thatscene description.

Upon completion of image sensing, the composition process is executed toform one moving image stream file. Then, a message that prompts the userto connect his or her portable terminal is displayed on the displayscreen, and when the user inputs a transfer instruction at the console103, the file begins to be transferred to the user's terminal.

To implement the aforementioned process, the storage unit 105 pre-storesand manages various objects as a database, which has a structure shownin, e.g., FIG. 14.

That is, stories are managed in the uppermost layer, and scenedescription files, and selectable background and feature player objectsare hierarchically managed under the corresponding stories. When “golflessen” is selected, as described above, a corresponding scenedescription file is selected, and the selectable background scene andfeature player are selected from those depending on the selected story.

When the user has a portable terminal comprising components 711 to 717(corresponding to 501 to 507 in FIG. 5), he or she can connect theterminal to a communication unit of the amusement apparatus via acomponent 718 (wired/wireless communication means). In this way, theuser can receive objects generated by the amusement apparatus by his orher portable terminal, can save them as a file, and can play it back asneeded.

As shown in FIG. 8, the portable terminal can communicate with anotherportable terminal or a computer (to be referred to as a PC hereinafter)that can execute a process to be described later. The PC can exchangeobjects with another portable terminal via communications, andcomposites/plays back a moving image as long as it can execute theprocess shown in FIG. 6. Furthermore, the PC can generate new BIFS orcan edit the existing BIFS corresponding to objects or a story. Suchobject generation/edit process can be implemented by a CG/animationeditor or the like, and BIFS generation/edit process can be implementedby a normal editor or the like although BIFS knowledge is required.Also, tools for easily generating/editing objects, BIFS, and the likemay be delivered as a software package via a CD, software downloading,or the like. In this way, the user can enjoy a moving image using anoriginal story or objects.

In the above embodiment, the portable terminal has been explained as adestination of information generated by the apparatus of the presentinvention. Alternatively, a storage medium (e.g., a memory card) may bethe destination.

Second Embodiment

In the above embodiment, the user must possess a portable terminal. Inthe second embodiment of the present invention, moving image informationcontaining an image of the user can be provided even if the user doesnot possess any special device.

In the second embodiment, a file generated by compositing a plurality ofobjects is sent to, e.g., the mail address of the user. As a result, theuser can receive that mail at home, and can play it back.

The apparatus can be realized when the communication unit 109 in FIG. 1comprises an arrangement for connecting the Internet.

The user must input his or her own email address in addition to theaforementioned operation. FIG. 15 shows an email address input windowdisplayed on the display 104. When the user operates the console 103 toinput his or her own email address and then operates a confirmation key(not shown), the generated moving image file is sent to the emailaddress of that user. Note that the mail address may be input beforestep S202 or step S207 in FIG. 2, or the like. In step S207, the file issent to the designated email address.

In some cases, the user may input a plurality of email addresses. Inthis case, the user can send the file to his or her friends.

Note that the second and first embodiments may be combined. In thiscase, a message for prompting the user to select an email address or theuser's portable terminal as a destination is displayed, and the file canbe output in the procedure of the first or second embodiment accordingto the selected contents.

Moreover, the function of the potable terminal mentioned above can beadded to a cellular phone having e-mail function, it becomes convenientfor the user.

Third Embodiment

In the first and second embodiments, a composite moving image containinga sensed image of the user is output to the user's portable terminal ordesignated email address. Alternatively, the moving image generated bythe apparatus of the present invention may be uploaded to a specificserver on the Internet.

That is, moving image files generated by the amusement apparatus of thepresent invention are managed by a single agent. However, if manyunspecified persons can access such files via the Internet, it is notpreferable in terms of privacy. To solve this problem, upon completionof sensing of an image of the user, a URL upon browsing and a receiptnumber are issued, and a message for prompting the user to input apassword is displayed on the display screen. Then, the generatedcomposite moving image file, receipt code, and password are uploaded tothe specific server to ask it to manage them.

In this case, the step of informing a receipt code and the step ofinputting a password may be inserted immediately before step S207, and afile and the like may be sent to a predetermined server in step S207.

As a result, as shown in FIG. 9, a server 901 stores and manages movingimage files transferred from respective amusement apparatuses, and theuser accesses the server 901 using his or her portable terminal 902 or903. When the user accesses the server, an input window of the receiptnumber and password is displayed, and after authentication, thecorresponding file is transferred to the user using, e.g., an FTPprotocol.

Therefore, when the user informs his or her friends of the URL, receiptcode, and password via some means (email or phone message), he or shecan give his or her friends an opportunity to observe user's movingimage. Note that the server 901 may require the user payment of aregistration/management fee of objects and sales commission.

Fourth Embodiment

In the above embodiments, various objects are pre-stored in the storageunit 105 of the amusement apparatus. However, in order to prevent theuser from wearying, data of background scenes, feature players, and thelike are preferably added and updated appropriately. In the fourthembodiment, the communication unit 109 used in the second and thirdembodiments accesses a specific server, e.g., once a week to downloaddata of new background scenes, feature players, and scene descriptionfiles, if they are available, and to register them in the database ofthe storage unit 105.

In this case, the controller 110 can execute a process shown in FIG. 16.

In step S1601, the amusement apparatus connects a server (a servermanaged by the management company of the amusement apparatus of thepresent invention) that manages objects on the Internet via thecommunication unit 109. It is then checked in step S1602 if the serverstores a new object (scene description file, background scene, featureplayer). If NO in step S1602, the connection is disconnected in stepS1603, and this process ends.

On the other hand, if it is determined that a new object is available,the flow advances to step S1604 to download that object. The downloadedobject is registered in the database of the storage unit 105 in stepS1605, and the connection is disconnected in step S1606, thus endingthis process.

Note that object types include scene description files, backgroundscenes, and feature players, and can be discriminated from each other bydescribing the type in their files. For example, upon registering anobject of “feature player X”, the position of the tree structure shownin FIG. 14 where the object is to be registered must be determined. Inthis embodiment, that position can be determined by a storagedestination described in the corresponding file. Also, otherdetermination methods may be used.

Fifth Embodiment

In the first to fourth embodiments, since a moving image is generated asdigital data, and objects to be composited may be copyrighted in somecases, it is desired to prevent illicit copies.

Hence, the fifth embodiment will exemplify a measure taken against suchcopies. For the sake of simplicity, the fifth embodiment will beexplained in correspondence with the first embodiment, but may beapplied to any of the second to fourth embodiments.

The arrangement of the amusement apparatus is substantially the same asthat shown in FIG. 1, except that a copyright protection unit 100 isadded, as shown in FIG. 17. The operation processing sequence of thisembodiment will be described below using FIG. 18.

The user inserts a coin or coins into the insertion port 101 (stepS1801). Note that the present invention is not limited to coin, but acredit card, or an IC card, prepaid card, or the like having a functionof electronically receiving money may be used.

Upon detection of coin insertion, the display 104 displays storiesprepared in advance in a storage unit 105 (e.g., a storage device suchas a hard disk, MO, CD, DVD, or the like) as some selectable menu itemsvia the processor 106 (to be described later), and the user selects adesired story at the console 103 (step S1802).

After the story is selected, some background scenes, feature players,and the like (objects) corresponding to that story, which are stored inthe storage unit 105, are displayed on the display screen of the display104 via the processor 106, and the user selects a desired backgroundscene, feature player, and the like at the console 103 (step S1803). Inplace of displaying the stories, background scenes, and the like, theymay be announced by voice using the speaker (or earphone) 107, and theuser may input his or her choice via the microphone 108.

After the story, background scene, feature player, and the like aredetermined, the motion and voice of the user corresponding to the storyare captured as digital moving image information using the camera 102and the microphone 108. The captured digital information is extractedand compressed by the processor 106 (step S1804).

The processor 106 composites the compressed sensed information with theselected background scene and feature player and plays back theinformation using the display 104 and speaker 107 (step S1805). Theapparatus prompts the user to check if the playback information is OK.If the user inputs a re-input instruction at the console 103, theprocess can be repeated from step S1802, S1803, or S1804 (step S1806).Note that this repetition process may be omitted, and the process may bedone only once. Alternatively, the flow may return to step S1801, andthe user may select a repetition process by inserting extra money.

If the user inputs a confirmation instruction of the playbackinformation, he or she then inputs an instruction for executing acopyright protection process for individual objects at the console 103(step S1807). In this process, when the story or objects selected insteps S1802 and S1803 are specially set to protect their copyrights, anextra charge process for these copyrights, a process for setting a copycount corresponding to the extra charge upon copying or transmittingobjects, an encryption process of objects and the like are done. Theextra charge process for copyrights can be implemented by displayingcomments that pertain to copyrights, their use fees, and the like on thedisplay 104, calculating the amount of extra money to be paid by theuser, and prompting the user to insert the extra fee from the insertionport 101. In this embodiment, in order to simplify the processing flow,processes associated with copyright protection are executed together instep S1807, but may be distributed to processes in steps S1801 to S1806and S1808.

For example, upon selecting a story or objects in steps S1802 and S1803,a display process of a message indicating that the story or object theuser is about to select is copyrighted, and an extra charge process maybe done. A charge process by coin insertion in step S1801 may be donesimultaneously in this step in place of the first step of the flow. Acopyrighting process is done by the copyright protection unit 100, andits example will be explained later. Finally, the playback informationis sent via the communication unit 109 (step S1808).

In the above description, the respective units 101 to 109, and theoperation shown in FIG. 18 are controlled by the controller 110 whichcomprises a CPU, a ROM/RAM for storing programs, and the like.

Note that a method of compositing a plurality of objects is based onMPEG-4 as in the first embodiment, and a detailed description thereofwill be omitted.

When object data are copyrighted, an MPEG-4 player must limit playback.In MPEG-4, a mechanism called IPMP (Intellectual Property Management &Protection) has been proposed for copyright protection, and thisembodiment uses this mechanism.

FIG. 19 shows a schematic arrangement of a general MPEG-4 player. Atransmission path 401 is a data path such as various networks, computerbus, and the like, and is a network to which an MPEG-4 stream is input.Note that the transmission path 401 also means an interface between theplayer and a recording medium device such as a CD-ROM, DVD-ROM, DVD-RAM,or the like in addition to the communication path. In the player, anMPEG-4 stream delivered from the network or played back from therecording medium device is input to a demultiplexer 402. The MPEG-4stream is demultiplexed by the demultiplexer 402 into scene descriptiondata, moving image (video) object data, audio object data, objectdescription data, and the like, which are respectively input tocorresponding sync layer blocks (buffer memories) 403. Note that theaudio object data has undergone high-efficiency (compression) codingsuch as known CELP (Code Excited Linear Prediction) coding, transform-domain weighted interleave vector quantization (TWINVQ) coding, orthe like. Also, the moving image object data has undergonehigh-efficiency coding by, e.g., known MPEG-2, H-263, or the like.

The respective object data of the sync layer blocks 403 are input tocorresponding decoders 404 to 407. The decoders 404 to 407 decode theaforementioned scene description data, moving image object data, audioobject data, and the like which have undergone high-efficiency coding.In FIG. 19, this embodiment assumes an apparatus which can decode theseobjects even when the MPEG-4 bitstream contains a plurality of differenttypes of objects in each of audio objects, moving image objects, andobject description data. For this reason, a plurality of sets of synclayer blocks and decoders 404 to 407 are prepared in correspondence withaudio objects, moving image objects, and object description data, butthey can be modified depending on the systems used.

The audio objects, moving image objects, and object description datadecoded by the decoders 405 to 407 undergo a composition/graphic processin a scene composition unit 408 on the basis of the scene descriptiondata decoded by the scene description decoder 404. A final data sequenceobtained in this way is supplied to an output device 409 such as adisplay, printer, or the like, and is visualized and made audible.

When individual object data such as audio objects, moving image objects,and the like that form a scene must undergo control for executing orstopping playback for copyright protection, IPMP information is used.The IPMP information is transferred using an IPMP stream as a buildingcomponent of the received data stream. An IPMP controller 411 interceptsbitstreams at control points 410 or accesses the decoders 405 to 407 tocommand them to stop decoding as needed on the basis of IPMP informationcontained in the IPMP stream from the demultiplexer 402. For thisreason, if it is determined based on the IPMP information that the useris not authorized to view the data, since data cease to be decoded,playback stops. With this control, copyrighted data are protected.

Copyright protection of the copyright protection unit 100 is done asfollows. As described above, individual objects to be played back arecontrolled by the IPMP information. Hence, copyrighted objects can beprotected by the IPMP information.

Furthermore, the copy count or transmission count of each object can becontrolled as follows. Control of the copy count or transmission counthas not been standardized yet in the IPMP information. Hence, control ismade using the following mechanism.

COP (Copy & Output Protection) information for controlling the copycount and transmission count is appended to each object in addition tothe IPMP information, and upon transmitting an object, the communicationunit 109 is controlled by the COP information. For example, when the COPinformation describes an allowable transmission count which is not zero,an object is allowed to be transmitted, and the allowable transmissioncount is decremented by 1 to update the COP information. If theallowable transmission count is zero, the operation of the communicationunit 109 is stopped, or the storage unit 105 is controlled to stop readof that object therefrom. When the multiplexer 306 is used, it iscontrolled to exclude that object from those to be multiplexed. Thedefault value and updated value of the COP information can be setdepending on money information obtained by detecting money inserted fromthe insertion port 101.

Therefore, the copyright protection unit 100 can comprise units (IPMPinformation generator 1101 and COP information generator 1102) forgenerating IPMP information and COP information according topredetermined formats. If future standardization of MPEG-4 allows tocontrol the copy count or playback information, the same process can beimplemented using that information. Also, the IPMP information generator1101 and COP information generator 1102 are controlled by the controller110.

Also, objects may be encrypted for copyright protection. In this case,the copyright protection unit 100 can comprise an encryption unit 1104.Key information used to decrypt the encrypted object can be generated bya key information generator 1103, and can be managed and exchangedtogether with the IPMP information and COP information.

With the aforementioned arrangement of the moving image output apparatusand its control method, objects having copyrights and/or rights oflikeness can be protected, and a scene containing the protected objectscan be generated.

FIG. 20 is a block diagram of a portable terminal (moving-image playbackapparatus) as the user's terminal, and FIG. 21 shows its operation flow.The difference from the arrangement in FIG. 5 is that a copyrightprotection unit 508 is added. Note that program codes according to theflow chart of FIG. 21 are stored in a storage unit 505 (to be describedlater), and are loaded onto a RAM (not shown) and executed uponoperation of the portable terminal. The portable terminal has a functionof exchanging generated objects, and a function of allowing the user toindividually enjoy contents by compositing/playing them back.

The portable terminal of this embodiment can display some processingmenu items on a display 504, and the user can select a process using aconsole 503. The menu includes a process for sending/receiving objects,a process for compositing/playing back a moving image, a process forchanging each object, and the like. The menu can also include an itemthat pertains to copyright protection.

Referring to FIG. 21, assume that the user selects atransmission/reception process (step S2105) in the process select step(step S2102) after the process starts (step S2101). In this process,objects are received from a moving image output apparatus(aforementioned amusement apparatus) or another portable terminal, orobjects are sent to another terminal using a communication unit 501.

A case will be examined below wherein objects (containing BIFScorresponding to a story) generated by another apparatus or terminal arereceived. A transmission/reception instruction is issued at the console503, and objects received via the communication unit 501 are stored inthe storage unit 505 (in the transmission process, objects which arestored in the storage unit 505 and are designated using the display 504or console 503 are sent to another portable terminal via thecommunication unit 501). In order to confirm objects to betransmitted/received, the video and audio of those objects may be playedback via a processor 506 (to be described later), and may be stored ortransmitted after confirmation. Prior to such transmission/reception, acopyright protection process (step S2110) is executed. In this copyrightprotection process, if an object to be transmitted is copyrighted, andespecially has a limited transmission count, it is controlled by theaforementioned COP information. That is, the object is transmitted whenthe allowable transmission count in the COP information is not zero, andthe allowable transmission count is decremented by 1 to update the COPinformation. When the allowable transmission count is zero, theoperation of the communication unit 501 is stopped or the storage unit505 is controlled to stop its input The copyright protection processthat pertains to the COP information is executed by the copyrightprotection unit 508.

A case will be examined below wherein a composition/playback process(step S2104) is selected. In this process, objects which are stored inthe storage unit 505 and form the story are played back using thedisplay 504 or speaker 502 via the processor 506. Note that theprocessor 506 can be implemented by the scene composition unit 1001 inFIG. 10. When an object to be played back is to undergo playback controlby IPMP information, encryption, or the like, the processor 506 iscontrolled by the copyright protection unit 508 (step S2109).

A case will be examined below wherein an object change process (stepS2103) is selected. In this process, when new objects are added via acommunication with another terminal in step S2105, objects in one storystored in the storage unit 505 are replaced by these new objects, or thenew objects are added to that story. Such process is implemented byrewriting information called a descriptor which indicates the attributeof each object or information described in BIFS within an allowablerange. In the object change process as well, if the object is protectedby IPMP or the like, the processor 506 that executes the aforementionedprocess is controlled by the copyright protection unit 508 (step S2108).

Whether or not a series of processes mentioned above is to end isconfirmed, and another process can be repeated (step S2106). Forexample, after some objects are received from the amusement apparatus ofthe embodiment mentioned above or another portable terminal and arestored by the transmission/reception process in step S2105, the processmay proceed to confirm the contents of the received objects by selectingthe composition/playback process in step S2104. Furthermore, thecomposition/playback process in step S2104 may be repeated whileexecuting the object change process in step S2103. Finally, the processends. If it is determined in the copyright protection process in stepS2108, S2109, or S2110 that the process cannot proceed, the flow skipsthe next process, and returns to process selection in step S2102 toexecute another process.

When information received from the amusement apparatus or the like is anMPEG-4 stream multiplexed by the multiplexer 306 in FIG. 3, theprocessor 506 may include the demultiplexer 402. The outputs from thedemultiplexer are supplied not to the sync layer blocks but to thestorage unit 505. A controller 507, which comprises a CPU, a ROM/RAM forstoring programs, and the like, controls the respective units 501 to506, and the operation shown in FIG. 21.

The copyright protection unit 508 comprises a COP information controller1202 for updating/controlling COP information, and an IPMP controller1201 for controlling objects in accordance with IPMP information, asshown in FIG. 23. When objects are encrypted, the unit 508 may alsocomprise a decryption unit for decrypting encrypted objects using keyinformation received together with the COP information and IPMPinformation.

With the aforementioned arrangement of the portable terminal and itscontrol method, a scene containing objects, copyrights or rights oflikeness are protected, can be played back.

Also, a moving image processing system constructed using some of themoving image output apparatus, the portable terminal (moving imageplayback apparatus) of this embodiment, and a personal computer (to bereferred to as a PC hereinafter) having functions to be described latercan be applied to FIGS. 7 to 9. Also, such system can be applied to thesecond to fourth embodiments.

With the aforementioned moving image processing system and its controlmethod, an object edit system that exchanges objects among systems,edits them, and exchanges edited objects again can be built.

Another Embodiment

The objects by means of the apparatuses and their control methods in theabove embodiments are achieved not only by a method of combining theapparatuses and their control method but also by supplying a programcode of software that can implement the functions of the above-mentionedembodiments to a computer (or a CPU or MPU) in the system or apparatus,and making the computer control various devices in the system orapparatus.

In this case, the program code itself read out from the storage mediumimplements the functions of the above-mentioned embodiments, and theprogram code itself, and means for supplying the program code to thecomputer (i.e., a storage medium which stores the program code) areincluded in the scope of the embodiments.

As the storage medium for storing the program code, for example, afloppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM,magnetic tape, nonvolatile memory card, ROM, and the like may be used.

The program code is included in the scope of the embodiments not onlywhen the functions of the above embodiments are implemented bycontrolling various devices according to the supplied program code alonebut also when the functions of the embodiments are implemented bycollaboration of the program code and an OS (operating system) oranother application software running on the computer.

Furthermore, the program code is included in the scope of the presentinvention when the functions of the above-mentioned embodiments areimplemented by some or all of actual processing operations executed by aCPU or the like arranged in a function extension board or a functionextension unit, which is inserted in or connected to the computer, afterthe supplied program code is written in a memory of the extension boardor unit.

As described above, according to the above embodiments, an amusementsystem designed for moving images can be easily built. Also, movingimage information obtained by compositing a moving image of the userhimself or herself and objects selected by the user can be generated.

Since the above embodiment uses MPEG-4 as a compression/transmissionscheme of moving images, a moving image amusement system which canreduce the storage size, can easily composite moving images, allows theportable terminal to easily use a moving image, and can easily exchangedata in units of objects can be built. Using MPEG-4, the informationsize can be reduced, and information can be provided by sending it tothe user's terminal or the network address of, e.g., the Internet, thusallowing the user to freely use that information.

Furthermore, since the information to be provided has informationindicating use conditions of objects, object copyrights can be protectedand their copy counts or the like can be limited. As many apparentlywidely different embodiments of the present invention can be madewithout departing from the spirit and scope thereof, it is to beunderstood that the invention is not limited to the specific embodimentsthereof except as defined in the appended claims.

1-35. (canceled)
 36. A moving image playback apparatus comprising: inputmeans for inputting stream data generated by multiplexing a plurality ofobject data; demultiplexing means for demultiplexing the stream datainto the plurality of object data; playback means for playing back theobject data demultiplexed by said demultiplexing means; detection meansfor detecting COP (Copy & Output Protection) information of each objectdata demultiplexed by said demultiplexing means; and control means forcontrolling a playback operation of said playback means in accordancewith a detection output of said detection means.
 37. A portable terminalcomprising a moving image playback apparatus cited in claim
 36. 38-39.(canceled)
 40. A control method of a moving image playback apparatus,comprising: an input step of inputting stream data generated bymultiplexing a plurality of object data; a demultiplexing step ofdemultiplexing the stream data into the plurality of object data; aplayback step of playing back the object data demultiplexed in thedemultiplexing step; a detection step of detecting COP (Copy & OutputProtection) information of each object data demultiplexed in thedemultiplexing step; and a control step of controlling a playbackoperation of the playback step in accordance with a detection output ofthe detection step. 41-42. (canceled)
 43. A storage medium for storing aprogram code which functions as a moving image playback apparatus forplaying back stream data, comprising: a program code of an input step ofinputting stream data generated by multiplexing a plurality of objectdata; a program code of a demultiplexing step of demultiplexing thestream data into the plurality of object data; a program code of aplayback step of playing back the object data demultiplexed in thedemultiplexing step; a program code of a detection step of detecting COP(Copy & Output Protection) information of each object data demultiplexedin the demultiplexing step; and a program code of a control step ofcontrolling a playback operation of the playback step in accordance witha detection output of the detection step.
 44. A moving image outputapparatus comprising: storage means for storing various object data in ascene; input means for inputting request data indicating a request ofdesired object data stored in said storage means; and output means foroutputting the object data and information data that pertains to a usecondition of the object data in response to the request data.
 45. Theapparatus according to claim 44, wherein the information data is IPMP(Intellectual Property Management & Protection) information.
 46. Theapparatus according to claim 44, wherein the information data is COP(Copy & Output Protection) information.
 47. A control method of a movingimage output apparatus, comprising: a storage step of storing variousobject data in a scene in storage means; an input step of inputtingrequest data indicating a request of desired object data stored in thestorage means; and an output step of outputting the object data andinformation data that pertains to a use condition of the object data inresponse to the request data.
 48. A storage medium for storing a programcode which functions as a moving image output apparatus for outputtingobject data, comprising: a program code of a storage step of storingvarious object data in a scene in storage means; a program code of aninput step of inputting request data indicating a request of desiredobject data stored in the storage means; and a program code of an outputstep of outputting the object data and information data that pertains toa use condition of the object data in response to the request data. 49.A potable terminal comprising: reception means for receiving code dataof an moving image having been coded in units of objects; storing meansfor storing the code data received by said reception means; decode meansfor decoding the code data stored by said storing means; reproducingmeans for reproducing the moving image by composing each object decodedby said decode means; and transmitting means for transmitting the codedata to an external device.
 50. The terminal according to claim 49,wherein said terminal has also telephone function.